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Abstract 

In a “publish-or-perish culture”, the ranking of scientific journals plays 
a central role in assessing performance in the current research environ¬ 
ment. With a wide range of existing methods and approaches to deriv¬ 
ing journal rankings, meta-rankings have gained popularity as a means 
of aggregating different information sources. In this paper, we propose 
a method to create a consensus meta-ranking using heterogeneous jour¬ 
nal rankings. Using a parametric model for paired comparison data we 
estimate quality scores for 58 journals in the OR/MS community, which 
together with a shrinkage procedure allows for the identification of clusters 
of journals with similar quality. The use of paired comparisons provides 
a flexible framework for deriving a consensus score while eliminating the 
problem of data missingness. 


1 Introduction 

While the so-called “publish-or-perish culture” has been widely discussed and 
criticized mainly because of intense publication pressure (see for example Adler 
and Harzing, 2009; Frey, October 19. 2010; Willmott, 2011), an increasingly 
competitive research environment is in need of performance metrics. Gener¬ 
ally, the reputation and quality of the research outlets in which scholars publish 
their work together with received citations in peer-reviewed academic journals 
remain the main criteria for assessing research quality. The main research out¬ 
lets and means of knowledge dissemination in each business discipline are the 
academic journals in which its research is published (Meredith et ah, 2011; Fry 
and Donohue, 2013). In an attempt to reflect the impact and quality of journals, 
rankings therefore lie at the core of research assessment. Whether it is used by 
universities for hiring or promotion/tenure purposes, by editors to underline 
the importance of their journals or by publishers who aim at maximizing their 
revenue, a ranking supports decision making. Especially due to this importance 
in academic life the number of applied methods to create rankings is increasing 
continuously. 
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While there are many different rankings available, it is often hard to pick the 
most appropriate ranking for a special purpose or a certain institution. Differ¬ 
ent stakeholders often prefer a different ranking for various reasons, such that 
the creation of meta-rankings is the only way to comfort all involved parties. 
Moreover, the disadvantages of the conventional approaches to creating jour¬ 
nal rankings, with citation-based and survey-based approaches being the most 
popular, have been outlined in the literature (i.e., Serenko and Dohan, 2011). 
In this light, meta-rankings have gained popularity due to their the advantage 
of building on previous work and aggregating the available information rather 
than building against it. 

In this paper, we propose a novel method to aggregate heterogeneous 
rankings using adaptive lasso estimators and apply the method to obtain 
a consensus meta-ranking for 58 established journals in the Operations Re¬ 
search/Management Science (OR/MS) community. 

The parametric Bradley-Terry model (Bradley and Terry, 1952) for paired 
comparison data is employed for estimating journal quality scores as an alter¬ 
native to the popular data envelopment analysis (DEA) method (used in i.e., 
Fry and Donohue, 2013; Tiiselmann et al., 2015). 

The use of paired comparisons provides a flexible framework for deriving a 
consensus score using different sources, with no strict restriction on the number 
of journals and the number of rankings used in the analysis. Together with a 
shrinkage procedure, the proposed approach allows the identification of clusters 
of journals with similar quality (Masarotto and Varin, 2012). The variability of 
the parameter estimators is determined using the parametric bootstrap. 

We also investigate whether the consensus meta-ranking changes significantly 
when using only information from rankings published in 2013 and find that 
results remain mainly stable, with few significant changes. 

Tiiselmann et al. (2015) discuss several shortcomings of meta-studies, among 
which arbitrary inclusion or datedness of journal rankings and journals, inade¬ 
quate treatment of missing data and treatment of ordinal data as metric. We 
address these shortcomings in the following way: (i) we base our analysis on an 
extensive list of 31 journal rankings that are used by stakeholders in decision 
making. An overview of the rankings is provided in Section 3 and our descrip¬ 
tive statistics show that the significant correlations at a 5% significance level are 
positive. However, each of the rankings contains valuable information because, 
while having the common goal of assessing journal quality, they proceed differ¬ 
ently in defining comparison measures, (ii) Our approach eliminates the evident 
problem of missingness in journal lists, while existing data need not be disre¬ 
garded due to the use of pairwise comparisons between journals as expressed 
by the various ranking lists (similar to Cook et al., 2010; Thcu/31 et al., 2014). 
(iii) The ordinal nature of rankings is kept by the pairwise comparison data. In 
addition, the parametric model is adapted to appropriately handle ties. 

This paper is organized as follows. In Section 2 we present an overview of 
existing ranking approaches. Section 3 presents the journals and journal ranking 
lists used in the study, Section 4 introduces the methodology used for creating 
a consensus journal rating. The results are discussed in Section 5 and Section 6 
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concludes. 


2 Ranking approaches 

Even if quality is abstract by nature, several approaches have been proposed 
for assessing journal quality. Survey-based rankings are primarily created by 
universities or associations, who ask scholars with different affiliations to rate 
journals on a usually ordinal scale. Academic research relying on survey-based 
quality assessment of Operations Research, Management Science, Production 
& Operations Management (OR/MS/POM) journals has been conducted by 
Barman et al. (1991, 2001) or Olson (2005), who surveyed faculty members of 
25 business schools in the US in 2000 and 2002. 

Citation-based metrics are mainly published by commercial providers like 
Thomson Reuters or Elsevier. The first academic studies to conduct a citation- 
based analysis of journals in the Operations Management (OM) field are Vokurka 
(1996) and Goh ct al. (1996). A recent citation-based analysis focused on the 
OR/MS field is Xu et al. (2011), who use Google’s PageRank method to rank 
31 OR/MS journals. The PageRank method is citation-based and differentiates 
citations by quality/source (i.e., citations from higher impact journals should 
outweigh citations from lower impact journals). This method is in essence simi¬ 
lar to the Eigenfactor® Score , which will be discussed briefly in Section 3. Hy¬ 
brid rankings are a combination of the subjective and objective approach (i.e., 
Zhou et al., 2001), while author-based approaches take into consideration also 
the author affiliation. Holsapple and Lee-Post (2010) perform a behaviour-based 
analysis, by examining the publishing behaviors of tenured OM researchers at 
leading research universities in the US. Extensive comparisons among the dif¬ 
ferent methods as well as shortcomings can be found in i.e., Donohue and Fox 
(2000), Frey and Rost (2010), Serenko and Dohan (2011). 

Considering the setbacks of the approaches mentioned above, the meta¬ 
ranking approach arises naturally as it tries to reconcile the variety of methods 
by using available information to build a composite journal ranking ((look et al., 
2010). In the OM field several meta-analyses have combined information from 
prior academic studies. Petersen et al. (2011) provide the first meta-analysis 
to examine the ranking of journals in the OM field by using 5 prior studies. 
Meredith et al. (2011) build journal rankings using official in-house journal lists 
of AACSB-accredited business schools and compare their results with 12 rank¬ 
ing studies published during 1990 to 2009. Fry and Donohue (2013) use the 
information from 15 previous OM journal ranking studies and provide a meta¬ 
analysis through DEA for assessing journal quality. A recent study is provided 
by Tuselmann et al. (2015), who base their study on 10 rankings from Harzing’s 
JQL data and the Impact Factor and compute an aggregate rating of journals 
in different subject areas by using DEA and random forests, with particular 
emphasis on OR/MS/POM. 
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3 Journals and quality assessment 

We select 58 journals established in the OR/MS community which are also part 
of Thomson Reuters’ Journal Citation Reports (Thomson Reuters, 2014). The 
complete list of journals can be found in Table 3. 

We collect an extensive list of 31 rankings still in use in 2013, both cita¬ 
tion and survey-based. We match the rankings to the journal list using ISSN 
codes. From Thomson Reuters’ Journal Citation Reports we obtain citation- 
based journal metrics like the Impact Factor, 5-Year Impact Factor, Immediacy 
Index and Cited Half-Life. The Impact Factor is one of the best known in¬ 
dexes and measures the number of citations received by a journal in the two 
preceding years normalized by the number of citable items in that journal. The 
5-Year Impact Factor extends the time period used for calculation to five years. 
The Immediacy Index measures the number of citations received by a journal 
in the same year, while Cited Half-Life measures the median age of a journal’s 
cited articles in a certain year. Other citation metrics freely available are the 
Eigenfactor® Score and its normalized version the Article Influence® Score. 
These metrics assign different weights to different sources of citations received 
by a journal and exclude self-citations. The time period used for calculation is 
five years 1 . 

From www.journalmetrics.com, the Source Normalized Impact per Paper 
metric (SNIP) corrects for differences between fields by dividing the number 
of citations per paper in a journal by the citable items in the journal’s subject 
field. The Impact per Publication (IPP) is a metric similar to Thomson Reuters’ 
Impact Factor and measures the average citations to peer-reviewed papers pub¬ 
lished in the three previous years. A third metric is the SCImago Journal Rank 
(SJR) which is in essence similar to the Eigenfactor® Score. Detailed method¬ 
ology can be found on the website. 

An important source of journal rankings (mostly survey-based) is the Harzing 
Journal Quality List (JQL). The 52nd edition of the JQL data (Harzing, 11 
February 2014) contains 22 rankings provided by different (research) institutions 
or scientific studies. Most of the JQL rankings are on an ordinal scale (see 
Table 4). 

All rankings used in the analysis are listed in Table 4. Out of 31 rankings, 
13 were compiled in 2013. The other 18 are dated in the range from 2001 to 
2012, but are still used in different contexts for assessing journal quality. 

Due to the (mostly) ordinal nature of the rankings, the presence of ties and 
incomplete journal lists, we assess the degree of association between the different 
sources by using the rank correlation coefficient proposed by Emond and Mason 
(2002), which corresponds to the unique association measure of Kemeny and 
Snell (1962) and satisfies four basic axioms that should apply to any distance 
measure between two weak orderings. The metric is denoted by t x and is an 
extension of Kendall’s r which handles ties more appropriately (see also Hornik 

1 See more on the methodology at http://www.eigenfactor.org/methods. 
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ct al., 2007): 



N C (N C - 1) 


where N c is the number of journals rated in at least one of the rankings Rk x 
and Rk 2 and atjk is defined as: 


1 if ranking R rates journal ,/, higher than or tied with as journal Jj, 


a ijk = s 0 if i = j or if R,k does not rate J, or/and Jj , 
I —1 otherwise. 


We present the t x correlation coefficients in Table 1, together with the num¬ 
ber of journals rated in the sample for each ranking. For each rank correlation 
coefficient, we compute p-values using 1000 bootstrap samples. Column “MLE” 
presents the correlation of the ranking produced using the method described in 
Section 4 with the other 31 rankings. The citation-based indexes of Thomson 
Reuters, the Eigenfactor® Score , the Article Influence® Score and the Elsevier 
metrics are mostly significantly correlated. Not surprisingly, the metric that is 
not significantly correlated with the rest is Cited Half-Life , as it is the only met¬ 
ric based on the age of cited articles. We observe that at 5% significance level 
the significant rank correlations are positive, but the degree of the association 
varies, indicating that the quality assessment of journals differs across different 
metrics or sources. 

We have in total 16452 pairwise comparisons between the 58 journals and 
each pair of journals has been on average compared 10 times. 
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p value < 0.001, ** - p value < 0.01, *- p value < 0.05. 





























4 Methodological approach 

Consider a total number of K journal rankings TZ = {Ri ,..., Rk} of N journals 
J = {Ji,..., Jn}- Each journal in the set J is ranked in at least one of the 
rankings in TZ. 

The aim of the analysis is suitably aggregating the information available 
from the K different ranking sources by finding a metric that reflects the overall 
quality of the journals in J. For each ranking Rk, we compare the journals 
included in the ranking pairwise by investigating whether a journal Ji has been 
ranked higher/lower than or the same as another journal Jj. Let Yijk be a 
variable taking one of the following values: 

{ 1 if ranking Rk rates journal Ji higher than journal Jj, 

0.5 if ranking Rk rates journal Ji and journal Jj the same, 

0 if ranking Rk rates journal Ji lower than journal Jj. 

A popular statistical model for modeling paired comparison data is the Bradley- 
Terry model (Bradley and Terry, 1952), which is a logistic regression for a binary 
outcome, where the probability of journal Ji being better than journal Jj is given 
by exp (fii — Hj)/{ 1 + exp(/x,; — fij)) with /.q and /ij being the ability (quality) 
parameters for journal Ji and journal Jj. Note that N need not be higher than 
K. 

As many journals in the analysis fall into the same ranking class, we employ 
a modification of the original Bradley-Terry model which can properly handle 
ties and treat ties as one-half of a success and one-half of a failure. The log- 
likelihood is then given by: 

N K 

%) = EE VijkitH - Ml) - log(l + exp (m - jij )). (1) 

i<j k 

The parameter of interest is the vector of journal abilities n = (pu,..., hn) T 
which can be further used for ranking the N journals. Only pairwise differences 
/^ — Hj are identifiable in model (1), hence a restriction needs to be imposed 
on the vector ^t. We choose to set the ability parameter of the first journal 
40R equal to zero. An alternative to treating ties as one-half of a success and 
one-half of a failure would be using the cumulative link Bradley-Terry model 
(Agresti, 2010). We choose the former due to reduced computational expenses. 

In order to avoid potential over interpretation of insignificant differences 
between journal abilities and obtain a clustering of the journals whose ability 
parameters are similar, Masarotto and Varin (2012) propose the ranking lasso 
technique, which computes the solution for the modified Bradley-Terry model 
by maximizing the likelihood and imposing a C\ penalty on the pairwise ability 
differences yu* — fij : 

( 2 ) 


N 


fix = arg min < + A ^ w tj \m - Hj \ 


i<j 
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where £(fi) is the log-likelihood function in (1), A is the lasso penalty and Wij 
are pair specific weights. We follow the suggestion of Vfasarotto and Varin 
(2012) and choose the weights inversely proportional to the maximum likelihood 
estimates (MLE): 

= | A f LE ) - |-i. 

This method is called the adaptive ranking lasso and it overcomes the problem of 
inconsistent non-zero lasso estimates due to the severe C\ penalty (Zou, 2006). 
The lasso penalty A can be chosen by minimization of an information criterion 
such as the Akaike Information Criterion (AIC). 

The uncertainty of the estimated abilities is evaluated using parametric boot¬ 
strap samples. Resampling from the modified Bradley-Terry model was per¬ 
formed by replacing the true parameters by their estimates. As adaptive lasso 
estimators (ALASSO) are by construction biased, we use the bias-corrected per¬ 
centile method (Efron, 1987) for computing 100(1 — 2a)% confidence intervals 
for the ability parameters. 


5 Results 

We proceed with presenting results from applying the method in Section 4 to 
the 58 journals and 31 rankings in Section 5.1. In Section 5.2 we repeat the 
analysis but use only rankings from 2013 and check the stability of the results. 

5.1 All rankings 

We compute the percentage of times each journal appeared in the rankings in 
Table 4. The journal that is rated most often is Operations Research (OR), fol¬ 
lowed by Production and Operations Management (POM) and Decision Support 
Systems (DSS) (see Table 2). Table 2 also presents the MLEs and the ALASSO 
estimators where the shrinkage parameter A was chosen according to AIC. The 
ALASSO estimators group journals with similar ability parameters and indicate 
24 clusters of journals. In addition, 95% bias-corrected confidence intervals are 
reported. The uncertainty in the ability parameters was estimated using 1000 
parametric bootstrap samples. An important feature that can be observed from 
the confidence intervals for the MLEs is that they are overlapping in some cases. 
This implies that some of the differences between the ability parameters are not 
significant and that the ranking lasso method should be employed for shrinking 
the differences of similar coefficients to zero. In most of the cases, the confidence 
intervals for the ALASSO estimators are shorter than for MLE, indicating that 
the shrinkage provides a reduction in variability. 

The correlation between the MLE and rankings used in the analysis is pos¬ 
itive (see column “MLE” in Table 1). The metric exhibiting the highest cor¬ 
relation with the MLE ranking of the modified Bradley-Terry model is the 
Eigenfactor® Score. 


Table 2: Maximum likelihood (MLE) and adaptive lasso estimators (ALASSO) for the modified Bradley-Terry model, with corre¬ 
sponding 95% bias-corrected percentile bootstrap confidence intervals 


Pos 

Journal 

% rated 

Rankings 2001-2013 
MLE 95% c.i. 

ALASSO 

95% c.i. 

Pos 

% rated 

MLE 

Rankings 2013 
95% c.i. 

ALASSO 

95% 

c.i. 

1 

MS 

83.9 

3.500 

(3.239, 3.753) 

2.980 

(2.900, 3.252) 

1 

76.9 

3.790 

(3.486, 4.144) 

2.730 

(2.648, 

3.159) 

2 

OR 

96.8 

2.420 

(2.193, 2.634) 

1.930 

(1.858, 2.176) 

8 

92.3 

2.050 

(1.791, 2.288) 

1.400 

(1.339, 

1.621) 

3 

MP 

64.5 

2.390 

(2.165, 2.604) 

1.930 

(1.846, 2.176) 

4 

61.5 

2.530 

(2.268, 2.790) 

1.680 

(1.586, 

1.904) 

4 

TRBM 

58.1 

2.340 

(2.119, 2.557) 

1.930 

(1.848, 2.176) 

3 

61.5 

2.780 

(2.470, 3.035) 

1.880 

(1.835, 

2.209) 

5 

JOM 

87.1 

2.300 

(2.093, 2.503) 

1.930 

(1.863, 2.176) 

2 

76.9 

2.850 

(2.586, 3.116) 

1.880 

(1.853, 

2.199) 

6 

TS 

61.3 

2.060 

(1.848, 2.298) 

1.680 

(1.568, 2.022) 

5 

69.2 

2.250 

(1.992, 2.491) 

1.400 

(1.339, 

1.606) 

7 

SCL 

35.5 

1.790 

(1.557, 2.023) 

1.310 

(1.225, 1.561) 

6 

46.2 

2.220 

(1.930, 2.466) 

1.400 

(1.349, 

1.606) 

8 

EJOR 

80.6 

1.650 

(1.438, 1.842) 

1.270 

(1.186, 1.491) 

7 

69.2 

2.050 

(1.831, 2.334) 

1.400 

(1.345, 

1.796) 

9 

TRELT 

64.5 

1.480 

(1.268, 1.687) 

1.040 

(0.973, 1.481) 

11 

92.3 

1.690 

(1.449, 1.902) 

1.200 

(1.061, 

1.499) 

10 

POM 

93.5 

1.400 

(1.205, 1.605) 

1.040 

(0.963, 1.351) 

10 

100.0 

1.750 

(1.518, 1.960) 

1.200 

(1.063, 

1.507) 

11 

MOR 

77.4 

1.320 

(1.120, 1.528) 

0.970 

(0.888, 1.187) 

16 

92.3 

1.210 

(0.994, 1.437) 

0.770 

(0.691, 

1.025) 

12 

DSS 

90.3 

1.300 

(1.098, 1.493) 

0.970 

(0.890, 1.202) 

14 

76.9 

1.310 

(1.095, 1.570) 

0.770 

(0.692, 

1.014) 

13 

MSOM 

64.5 

1.170 

(0.981, 1.394) 

0.870 

(0.776, 1.158) 

13 

100.0 

1.360 

(1.147, 1.580) 

0.770 

(0.693, 

1.079) 

14 

JOTA 

51.6 

1.160 

(0.959, 1.390) 

0.870 

(0.775, 1.158) 

12 

53.8 

1.450 

(1.194, 1.676) 

0.770 

(0.690, 

1.030) 

15 

OMEGA 

87.1 

1.150 

(0.947, 1.356) 

0.870 

(0.766, 1.186) 

9 

69.2 

1.980 

(1.742, 2.224) 

1.400 

(1.340, 

1.604) 

16 

JGO 

19.4 

0.920 

(0.686, 1.157) 

0.560 

(0.494, 0.944) 

18 

46.2 

0.970 

(0.732, 1.193) 

0.420 

(0.310, 

0.760) 

17 

HE 

80.6 

0.910 

(0.720, 1.108) 

0.560 

(0.501, 0.755) 

23 

84.6 

0.620 

(0.395, 0.831) 

0.370 

(0.234, 

0.662) 

18 

IJPR 

87.1 

0.870 

(0.685, 1.082) 

0.560 

(0.507, 0.812) 

19 

76.9 

0.890 

(0.668, 1.113) 

0.380 

(0.243, 

0.653) 

19 

JSCH 

61.3 

0.860 

(0.631, 1.054) 

0.560 

(0.504, 0.755) 

28 

84.6 

0.480 

(0.261, 0.698) 

0.370 

(0.241, 

0.625) 

20 

INFORMS 

61.3 

0.840 

(0.637, 1.053) 

0.560 

(0.499, 0.766) 

22 

84.6 

0.750 

(0.534, 0.956) 

0.380 

(0.230, 

0.628) 

21 

EXSA 

54.8 

0.740 

(0.535, 0.961) 

0.490 

(0.396, 0.754) 

20 

69.2 

0.810 

(0.580, 1.028) 

0.380 

(0.244, 

0.626) 

22 

AOR 

83.9 

0.710 

(0.503, 0.932) 

0.490 

(0.395, 0.769) 

21 

69.2 

0.760 

(0.521, 0.974) 

0.380 

(0.242, 

0.626) 

23 

COR 

77.4 

0.690 

(0.492, 0.891) 

0.490 

(0.398, 0.754) 

15 

69.2 

1.250 

(1.002, 1.472) 

0.770 

(0.697, 

1.134) 

24 

ORS 

64.5 

0.600 

(0.397, 0.802) 

0.450 

(0.302, 0.692) 

26 

69.2 

0.530 

(0.321, 0.760) 

0.370 

(0.241, 

0.625) 

25 

JORS 

83.9 

0.590 

(0.390, 0.806) 

0.450 

(0.305, 0.696) 

25 

69.2 

0.560 

(0.319, 0.803) 

0.370 

(0.239, 

0.653) 

26 

COA 

19.4 

0.570 

(0.337, 0.816) 

0.450 

(0.304, 0.708) 

24 

46.2 

0.610 

(0.348, 0.830) 

0.370 

(0.243, 

0.653) 

27 

ORL 

74.2 

0.510 

(0.318, 0.718) 

0.450 

(0.304, 0.683) 

32 

69.2 

0.110 

(-0.121, 0.334) 

0.000 

(-0.430 

, 0.000) 

28 

NSE 

29.0 

0.490 

(0.250, 0.723) 

0.450 

(0.297, 0.683) 

27 

69.2 

0.510 

(0.303, 0.743) 

0.370 

(0.242, 

0.653) 

29 

TNV 

74.2 

0.410 

(0.206, 0.601) 

0.380 

(0.072, 0.526) 

17 

69.2 

1.100 

(0.882, 1.320) 

0.720 

(0.607, 

0.935) 

30 

NETW 

38.7 

0.360 

(0.142, 0.587) 

0.380 

(0.071, 0.527) 

31 

53.8 

0.240 

(0.020, 0.490) 

0.270 

(0.000, 

0.488) 

31 

OMS 

29.0 

0.300 

(0.045, 0.526) 

0.380 

(0.063, 0.525) 

30 

69.2 

0.310 

(0.080, 0.538) 

0.270 

(0.000, 

0.498) 

32 

NAVRL 

71.0 

0.110 

(-0.086, 0.327) 

0.000 

(-0.459, 0.000) 

36 

92.3 

-0.200 

(-0.424, 0.012) 

-0.280 

(-0.540, 

-0.121) 

33 

JMS 

54.8 

0.040 

(-0.165, 0.254) 

0.000 

(-0.424, 0.000) 

29 

69.2 

0.330 

(0.104, 0.555) 

0.270 

(0.000, 

0.490) 

34 

FODM 

29.0 

0.010 

(-0.214, 0.234) 

0.000 

(-0.459, 0.000) 

33 

69.2 

0.010 

(-0.209, 0.228) 

0.000 

(-0.436 

, 0.000) 

35 

40R 

29.0 

0.000 


0.000 


34 

69.2 

0.000 


0.000 



36 

EO 

29.0 

-0.060 

(-0.281, 0.190) 

0.000 

(-0.455, 0.000) 

35 

69.2 

-0.060 

(-0.303, 0.148) 

0.000 

(-0.445 

, 0.000) 

37 

PPG 

80.6 

-0.230 

(-0.419, -0.023) 

-0.300 

(-0.624, -0.182) 

39 

76.9 

-0.380 

(-0.601, -0.134) 

-0.280 

(-0.544, 

-0.113) 



Table 2: (continued) 


Pos 

Journal 

% rated 

Rankings 2001 
MLE 95% 

-2013 

c.i. 

ALASSO 

95% 

c.i. 

Pos 

% rated 

MLE 

Rankings 2013 
95% c.i. 

ALASSO 

95% 

c.i. 

38 

QS 

35.5 

-0.280 

(-0.492, 

-0.040) 

-0.300 

(-0.629, 

-0.184) 

38 

46.2 

-0.290 

(-0.535, 

-0.060) 

-0.280 

(-0.538, 

-0.118) 

39 

OPT 

19.4 

-0.280 

(-0.520, 

-0.038) 

-0.300 

(-0.650, 

-0.183) 

37 

46.2 

-0.280 

(-0.496, 

-0.054) 

-0.280 

(-0.540, 

-0.118) 

40 

INTER 

77.4 

-0.310 

(-0.517, 

-0.091) 

-0.300 

(-0.561, 

-0.184) 

43 

53.8 

-0.630 

(-0.906, 

-0.396) 

-0.330 

(-0.546, 

-0.212) 

41 

OCAM 

19.4 

-0.420 

(-0.647, 

-0.176) 

-0.300 

(-0.633, 

-0.180) 

40 

46.2 

-0.420 

(-0.676, 

-0.206) 

-0.280 

(-0.544, 

-0.117) 

42 

OL 

29.0 

-0.480 

(-0.706, 

-0.265) 

-0.330 

(-0.655, 

-0.245) 

41 

69.2 

-0.500 

(-0.734, 

-0.271) 

-0.300 

(-0.561, 

-0.151) 

43 

ITOR 

64.5 

-0.550 

(-0.760, 

-0.323) 

-0.370 

(-0.658, 

-0.286) 

47 

53.8 

-0.840 

(-1.096, 

-0.582) 

-0.330 

(-0.536, 

-0.210) 

44 

DEDS 

19.4 

-0.620 

(-0.878, 

-0.373) 

-0.370 

(-0.658, 

-0.295) 

42 

46.2 

-0.620 

(-0.861, 

-0.380) 

-0.330 

(-0.562, 

-0.215) 

45 

DO 

29.0 

-0.620 

(-0.853, 

-0.401) 

-0.370 

(-0.657, 

-0.288) 

44 

69.2 

-0.640 

(-0.884, 

-0.424) 

-0.330 

(-0.562, 

-0.224) 

46 

IJITD 

19.4 

-0.700 

(-0.934, 

-0.438) 

-0.370 

(-0.658, 

-0.301) 

45 

46.2 

-0.710 

(-0.961, 

-0.468) 

-0.330 

(-0.562, 

-0.214) 

47 

OE 

29.0 

-0.780 

(-1.002, 

-0.542) 

-0.370 

(-0.658, 

-0.293) 

46 

69.2 

-0.800 

(-1.020, 

-0.561) 

-0.330 

(-0.546, 

-0.226) 

48 

MMOR 

67.7 

-0.920 

(-1.126, 

-0.706) 

-0.740 

(-1.102, 

-0.604) 

49 

84.6 

-1.230 

(-1.472, 

-0.986) 

-0.850 

(-1.173, 

-0.736) 

49 

TOP 

29.0 

-1.190 

(-1.410, 

-0.932) 

-0.890 

(-1.172, 

-0.813) 

48 

69.2 

-1.220 

(-1.481, 

-0.985) 

-0.850 

(-1.173, 

-0.723) 

50 

PJO 

29.0 

-1.230 

(-1.481, 

-0.980) 

-0.890 

(-1.172, 

-0.809) 

50 

69.2 

-1.260 

(-1.511, 

-1.032) 

-0.850 

(-1.053, 

-0.736) 

51 

CEJOR 

29.0 

-1.250 

(-1.501, 

-1.001) 

-0.890 

(-1.138, 

-0.826) 

51 

69.2 

-1.290 

(-1.554, 

-1.055) 

-0.850 

( — 1.141, 

-0.731) 

52 

INFOR 

16.1 

-1.410 

(-1.698, 

-1.114) 

-0.890 

(-1.222, 

-0.798) 

52 

38.5 

-1.430 

(-1.719, 

-1.152) 

-0.850 

(-1.090, 

-0.724) 

53 

JSIM 

22.6 

-1.650 

(-1.905, 

-1.359) 

-1.230 

(-1.498, 

-1.142) 

53 

53.8 

-1.690 

(-1.970, 

-1.410) 

-1.140 

(-1.339, 

-1.090) 

54 

ASMB 

29.0 

-1.790 

(-2.040, 

-1.521) 

-1.260 

(-1.530, 

-1.199) 

55 

69.2 

-1.830 

(-2.077, 

-1.561) 

-1.140 

(-1.364, 

-1.088) 

55 

RAIRO 

19.4 

-1.800 

(-2.082, 

-1.517) 

-1.260 

(-1.495, 

-1.217) 

54 

46.2 

-1.830 

(-2.108, 

-1.556) 

-1.140 

(-1.401, 

-1.072) 

56 

IMAJMM 

29.0 

-2.000 

(-2.264, 

-1.736) 

-1.390 

(-1.740, 

-1.346) 

56 

69.2 

-2.040 

(-2.329, 

-1.781) 

-1.270 

(-1.472, 

-1.222) 

57 

APJOR 

19.4 

-2.860 

(-3.207, 

-2.489) 

-2.120 

(-2.430, 

-2.038) 

57 

46.2 

-2.890 

(-3.285, 

-2.526) 

-1.980 

(-2.300, 

-1.900) 

58 

MILOR 

12.9 

-5.160 

(-6.653, 

-4.329) 

-3.320 

(-3.788, 

-3.243) 

58 

30.8 

-5.200 

(-6.649, 

-4.327) 

-3.050 

(-3.545, 

-2.986) 



Figure 1 illustrates the ALASSO estimators for each journal. Journals clus¬ 
tered together are marked by the same coloring. The journal with the high¬ 
est ability parameter is Management Science followed by Operations Research, 
Mathematical Programming (MP), Transportation Research Part B: Methodolog¬ 
ical and Journal of Operations Management (JOM). The ranking lasso groups 
the latter four journals into one cluster because of their similar MLEs. The 
journal with the lowest score is Military Operations Research (MILOR). 

Petersen ct al. (2011), Fry and Donohue (2013), and Tuselmann ct al. (2015) 
also identify Management Science as the top outlet for OR/MS/POM research. 
When regarding only the intersection of the sets of journals, JOM and POM 
(position 5 and 10 in the MLE ranking) also make it in the top ten lists of 
Meredith ct al. (2011) (position 1 and 2 respectively), Petersen et al. (2011) 
(position 3 and 4), Fry and Donohue (2013) (position 3 and 4). In Tuselmann 
et al. (2015), JOM is ranked second in the OR/MS/POM subject area while 
POM is on position 9. OR, MP, TS, EJOR are in the top ten of Fry and 
Donohue (2013), Tuselmann et al. (2015). IJPR receives a top ten position in 
Meredith ct al. (2011), Petersen ct al. (2011) and Fry and Donohue (2013), but 
in our analysis it is ranked only on position 18 (and position 11 in uselmann 
et al., 2015). 

5.2 Using 2013 rankings only 

An aspect that can be verified in our proposed framework is whether the new 
rankings (i.e., dated 2013) lead to a different consensus ranking from the one 
obtained using the whole sample of rankings. Rankings complied in 2013 could 
provide a more up-to-date picture of the current research environment. We ad¬ 
dress this topic be redoing the analysis using only the 13 rankings published in 
2013. The MLE and ALASSO estimates together with 95% confidence intervals 
are presented in Table 2. ALASSO estimators are visualized in Figure 2. The 
resulting journal ability parameters are similar to the ones using the whole sam¬ 
ple of 31 rankings and the consensus ranking remains highly stable. However, 
few differences can be identified. On the one hand, OR is downgraded from the 
second cluster to the fourth cluster and the estimated ability parameters have 
decreased significantly. Significantly reduced ALASSO estimators are observed 
also for ORL and NAVRL. On the other hand, JOM, OMEGA, TNV and COR 
have higher estimated ability parameters and higher positions in the ranking. 


6 Conclusion 

We use the ranking lasso method proposed by Masarotto and Varin (2012) to¬ 
gether with the Bradley-Terry model for paired comparison data in order to 
estimate quality scores for 58 OR/MS journals. The approach overcomes the 
issue of missing data by using pairwise comparisons and allows the identification 
of clusters of journals with similar quality scores. We conclude that Manage¬ 
ment Science maintains its position as lead OR/MS outlet, followed by OR, 
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Figure 1: Journal quality scores based on the whole sample of rankings - 
ALASSO estimators 
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Figure 2: Journal quality scores based on the rankings dated in 2013 - ALASSO 
estimators 
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MP, TRBM, JOM, TS, SCL, EJOR, TRELT and POM. The meta-analysis us¬ 
ing only 2013 rankings indicates increased quality for OMEGA and TNV. The 
presented method is flexible to integrate a wide range of heterogeneous rankings 
and the resulting graph provides a detailed outline on how to cluster and/or 
rate the underlying journals for subsequent usage for different purposes. 
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Table 4: Ranking lists. 


Abbrev Ranking Source Year Scale 


IF 1 

Impact Factor 

Thomson Reuters 

2013 

Numeric from 0. 

IF5Y 1 

5-Year Impact Factor 

Thomson Reuters 

2013 

Numeric from 0 

IMMI 1 

Immediacy Index 

Thomson Reuters 

2013 

Numeric from 0 

CHL 1 

Cited Half-Life 

Thomson Reuters 

2013 

Numeric from 0 

EFS 1 

Eigenfactor® Score 

University of Washington 

2013 

Numeric from 0 to 1 

AIS 1 

Article Influence® Score 

University of Washington 

2013 

Numeric from 0 

SNIP 2 

Source Normalized Impact per Paper 

Elsevier 

2013 

Numeric from 0 

IPP 2 

Impact per Publication 

Elsevier 

2013 

Numeric from 0 

SJR 2 

SCImago Journal Rank 

Elsevier 

2013 

Numeric from 0 

WieOl 

WU Wien Journal Rating 

WU Vienna University of Economics and Business 

2001 

D<C<B<A<A+ 

Vhb03 

VHB ranking 

Association of University Professors of Business in German speaking 
countries 

2003 

E<D<C<B<A< A+ 

Bjm04 

British RAE Rankings 

Geary et al. (2004) 

2004 

Numeric from 1 to 7 

Hkb05 

HKBU ranking 

Hong Kong Baptist University School of Business 

2005 

B- < B < B+ < A 

Theo05 

Theoharakis integrated ranking 

Theoharakis et al. index 

2005 

Numeric from 1.2 to 95 

Ejis07 

European Journal of Information Sys¬ 
tems ranking 

Mingers and Harzing (2007) 

2007 

1 < 2 < 3 < 4 

EjisCI07 

European Journal of Information Sys¬ 
tems ranking including citation impact 
factors 

Mingers and Harzing (2007) 

2007 

1 < 2 < 3 < 4 

UQ07 

University of Queensland Rating 

University of Queensland 

2007 

5<4<3<2<1 

Ast08 

Aston ranking 

Aston Business School 

2008 

1 < 2 < 3 < 4 

Wie08 

WU Wien Journal Rating 

WU Vienna University of Economics and Business 

2008 

A < A+ 

ABS10 

ABS Academic Journal Quality Guide 

Association of Business Schools 

2010 

1<2<3<4<4* 

Denll 

Danish Ministry Journal list 

Danish Ministry 

2011 

1 < 2 

HECll 

HEC Paris Ranking 

Hautes Etudes Commerciales de Paris 

2011 

C < B < B+ < A 

UQ11 

Adjusted ERA Rankings List 

University of Queensland 

2011 

4 < 3 < 2 < 1 

Vhbll 

VHB ranking 

Association of University Professors of Business in German speaking 
countries 

2011 

E<D<C<B<A< A+ 

Aeresl2 

AERES journal list 

Agence d’evaluation de la recherche et de l’enseignement superieur 

2012 

C < B < A 

Cral2 

CRA ranking 

Cranfield University School of Management 

2012 

1 < 2 < 3 < 4 

EJL12 

ERIM Journals Listing 

Erasmus Research Institute of Management 

2012 

S < P A < P < STAR 

Abdcl3 

ABDC Journal Rankings 

Australian Business Deans Council 

2013 

C < B < A < A* 

Cnrsl3 

CNRS ranking 

Centre National de la Recherche Scientifique 

2013 

4<3<2<1<1* 

Ess 13 

ESSEC journal ranking 

ESSEC Business School Paris 

2013 

3<2<1<0<0+ 

Fnegl3 

FNEGE ranking 

French Management Association 

2013 

4<3<2<1<1* 





1 based on Web of Science® database 

2 based on Scopus® database 



